Control System and Control Method

ABSTRACT

A control device estimates a position and pose of an imaging device relative to a robot based on an image of the robot captured by the imaging device. A simulation device arranges a robot model at a teaching point, and generates a simulation image of the robot model captured by a virtual camera that is arranged so that a position and pose of the virtual camera relative to the robot model in the virtual space coincide with the estimated position and pose of the imaging device. The control device determines an amount of correction of a position and pose of the robot for the teaching point so that the position and pose of the robot on the actual image captured after the robot has been driven according to a movement command to the teaching point approximate to the position and pose of the robot model on the simulation image.

BACKGROUND OF THE INVENTION Field of the Invention

The present technology relates to a control system and a control methodfor controlling a robot.

Description of the Background Art

Japanese Patent Laying-Open No. H11-239989 discloses a calibrationdevice which calibrates a simulation model of an object, using an actualimage of the object captured by a camera mounted on a robot.Specifically, a graphics image of the object that is generated havingthe same angle of view and viewing position as the camera and the actualimage of the object are superimposed one on the other and displayed, andthe simulation model is calibrated so that these images coincide witheach other.

SUMMARY OF THE INVENTION

The technology disclosed in Japanese Patent Laying-Open No. H11-239989presumes that the relative positional relationship is fixed between therobot and the camera mounted on the robot. Once a deviation occurs inthe relative positional relationship, the robot cannot be accuratelyoperated according to simulation. The technology disclosed in JapanesePatent Laying-Open No. H11-239989 is also not applicable to the casewhere the camera cannot be mounted on a robot.

The present invention is made in view of the above problem, and anobject of the present invention is to provide a control system and acontrol method which allows accurately operating a robot according tosimulation.

According to one example of the present disclosure, a control system forcontrolling a robot includes: a simulation device; a robot controller;an imaging device; and an estimation module. The simulation deviceperforms simulation using a robot model indicating a shape of the robot.The robot controller drives the robot according to a movement commandprovided to the robot controller. The imaging device captures an imageof the robot. The estimation module estimates a position and pose of theimaging device relative to the robot based on the image of the robotcaptured by the imaging device. The simulation device includes: asetting module for performing offline teaching to set a teaching pointindicative of a position and pose to be taken by the robot; and an imagegenerating module for generating a simulation image of the robot modelcaptured by a virtual camera arranged in a virtual space. The imagegenerating module arranges the robot model at the teaching point in thevirtual space and generates a first simulation image of the robot modelcaptured by the virtual camera that is arranged so that a position andpose of the virtual camera relative to the robot model coincide with theposition and pose of the imaging device estimated by the estimationmodule. The control system further includes a first acquisition moduleand a determination module. The first acquisition module provides therobot controller with a first movement command to the teaching point andacquires a first actual image of the robot captured by the imagingdevice after the robot has been driven according to the first movementcommand. The determination module determines an amount of correction ofa position and pose of the robot for the teaching point so that aposition and pose of the robot on the first actual image approximates toa position and pose of the robot model on the first simulation image.

According to the present disclosure, the actual operation of the robotcan be approximated to the operation of the robot model on the virtualspace simulated in the simulation device. In other words, an errorbetween the actual operation of the robot and the simulation operation,due to an error between the actual environment and the virtualenvironment, can be corrected automatically. As a result, the robot canbe accurately operated according to the simulation.

In the above disclosure, the determination module includes a deviationcalculation module and a correction amount calculating module. Thedeviation calculation module calculates a deviation between the positionand pose of the robot on an actual image of the robot captured by theimaging device and the position and pose of the robot model on the firstsimulation image. The correction amount calculation module calculates,as the amount of correction, an amount of movement from the position andpose of the robot that has been driven according to the first movementcommand to a position and pose of the robot when the deviation is lessthan a predetermined threshold.

According to the present disclosure, the actual operation of the robotcan be conformed to the simulation operation with more accuracy.

In the above disclosure, the determination module further includes asecond acquisition module for providing the robot controller with asecond movement command in a direction that reduces the deviation andperforming an acquisition process for acquiring a second actual imagecaptured of the robot by the imaging device when the robot has beendriven according to the second movement command. The second acquisitionmodule repeatedly performs the acquisition process since the robot hasbeen driven according to the first movement command until the deviationis less than the threshold. As the amount of correction, the correctionamount calculating module calculates a cumulative amount of secondmovement commands provided to the robot controller.

According to the present disclosure, the amount of movement from theposition and pose of the robot driven according to the first movementcommand to the position and pose of the robot when the deviation is lessthan the predetermined threshold, can be readily calculated.

In the above disclosure, the determination module includes a deviationcalculation module and a correction amount calculating module. Thedeviation calculation module calculates a deviation between the positionand pose of the robot on the first actual image and a position and poseof the robot model on the simulation image. The correction amountcalculating module calculates, as the amount of correction, an amount ofmovement from a position and pose of the robot model when the deviationis less than a predetermined threshold to a position and pose of therobot model arranged at the teaching point.

According to the present disclosure, there is no need to move the robotwhen determining the amount of correction of the position and pose ofthe robot, thereby reducing the time it takes to determine the amount ofcorrection.

In the above disclosure, the control system includes a control devicethat controls the robot controller and the imaging device, the controldevice being connected to the imaging device, the robot controller, andthe simulation device. The estimation module, the first acquisitionmodule, and the determination module are included in the control device.

According to the present disclosure, calculations necessarily for thedetermination of the amount of correction for the teaching point arecarried out in a distributed fashion among multiple pieces of equipment,including the control device, the robot controller, and the simulationdevice. In other words, loads on the equipment can be distributed.

According to one example of the present disclosure, a control systemthat controls a robot includes: a robot controller for driving the robotaccording to a movement command provided to the robot controller; and animaging device for capturing an image of the robot. A control method inthe control system includes first through fifth steps. The first stepperforms offline teaching using a robot model to set a teaching pointindicative of a position and pose to be taken by the robot. The secondstep estimates a position and pose of the imaging device relative to therobot based on the image of the robot captured by the imaging device.The third step arranges the robot model at the teaching point in avirtual space and generates a simulation image of the robot modelcaptured by the virtual camera that is arranged so that a position andpose of the virtual camera relative to the robot model coincide with theposition and pose of the imaging device estimated in the second step.The fourth step provides the robot controller with a movement command tothe teaching point and acquires an actual image of the robot captured bythe imaging device after the robot has been driven according to themovement command to the teaching point. The fifth step determines anamount of correction of a position and pose of the robot for theteaching point so that a position and pose of the robot on the actualimage approximates to a position and pose of the robot model on thesimulation image.

According to this present disclosure also, the robot can be accuratelyoperated according to simulation.

The foregoing and other objects, features, aspects and advantages of thepresent invention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram schematically showing an overall exampleconfiguration of a control system according to an embodiment of thepresent invention.

FIG. 2 is a diagram showing one example of a first actual image and afirst simulation image overlaid one on the other.

FIG. 3 is a block diagram showing an example hardware configuration of acontrol device according to the embodiment.

FIG. 4 is a schematic diagram showing an example hardware configurationof a simulation device according to the embodiment.

FIG. 5 is a block diagram showing an example functional configuration ofthe simulation device according to the embodiment.

FIG. 6 is a diagram illustrating a method of setting a teaching point.

FIG. 7 is a diagram illustrating a method of generation of a firstsimulation image corresponding to the n-th teaching point Pn.

FIG. 8 is a block diagram showing an example functional configuration ofthe control device according to the embodiment.

FIG. 9 is a diagram illustrating one example of a method of estimationof position and pose of the imaging device relative to a robot.

FIG. 10 is a diagram showing one example of changes in the actual imageas an acquisition process is performed.

FIG. 11 is a flow chart illustrating a flow of the first half of aprocess for determining an amount of correction of the position and poseof the robot.

FIG. 12 is a flow chart illustrating a flow of the last half of theprocess for determining the amount of correction.

FIG. 13 is a flow chart illustrating a flow of a part of a process fordetermining an amount of correction of the position and pose of therobot, according to Variation 1 of the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the present invention will be described, with referenceto the accompanying drawings. Note that the same reference sign is usedto refer to like or corresponding components in the drawings, anddescription thereof will not be repeated.

§ 1 Application

Initially, referring to FIGS. 1 and 2, one example scenario to which thepresent invention is applied will be described. FIG. 1 is a diagramschematically showing an overall example configuration of a controlsystem according to the present embodiment.

A control system SYS, illustrated in FIG. 1, includes a control device100, a robot 200, a robot controller 300, a simulation device 400, animaging device 500, and a camera controller 600.

Robot 200 is a mechanism which applies given processes (picking,machining, etc.) to an object. Robot 200, illustrated in FIG. 1, is avertical articulated robot. Robot 200 includes a base 21, an arm 22coupled to base 21, and an end effector 23 mounted on the distal end ofarm 22. Robot 200 further has a servomotor (not shown) for operating arm22.

Robot controller 300 performs trajectory calculation and anglecalculation for each axis according to a movement command from controldevice 100, and drives the servomotor, included in robot 200, accordingto a result of the calculations.

Simulation device 400 is configured of, for example, a general-purposepersonal computer (PC), and performs a simulation using a robot modelindicating the shape of robot 200. Specifically, simulation device 400performs offline teaching, thereby setting one or more teaching pointsindicating the position and pose to be taken by robot 200, andgenerating teaching data describing the set one or more teaching points.

The offline teaching is a process of displaying a robot model and anobject model, which indicates the shape of an object, on a virtualspace, and setting teaching points according to user input.

For example, when robot 200 performs a pick-and-place operation on anobject, multiple teaching points indicating positions and poses to betaken by end effector 23 are described in chronological order in theteaching data covering a period from a moment the end effector 23 graspsthe object until end effector 23 places the object on a given place.

Furthermore, simulation device 400 arranges a virtual camera on avirtual space and generates a two-dimensional simulation image of therobot model captured by the virtual camera.

As primary components, imaging device 500 includes an optical system,such as a lens, and an image sensor, such as a CCD (Coupled ChargedDevice) or CMOS (Complementary Metal Oxide Semiconductor) sensor.Imaging device 500 is installed so that robot 200 is within the field ofview of imaging device 500. Imaging device 500 captures two-dimensionalimage data (hereinafter, simply referred to as an “image”) of robot 200,in response to control by camera controller 600, and outputs thetwo-dimensional image data to camera controller 600.

Camera controller 600 controls the capturing by imaging device 500, inresponse to the image capture command from control device 100, andacquires the captured image from imaging device 500. Camera controller600 outputs the acquired image to control device 100. Note that cameracontroller 600 may apply given image processing (such as edge emphasisprocess) on the acquired image.

Control device 100 corresponds to an industrial controller whichcontrols a various objects, such as equipment and devices. Controldevice 100 is a type of computer that performs control calculations asdescribed below. Typically, control device 100 may be embodied as a PLC(programmable logic controller).

Control device 100 is connected to robot controller 300 and cameracontroller 600 via a field network, and exchanges data with robotcontroller 300 and camera controller 600. Furthermore, control device100 is also connected to simulation device 400 and exchanges data withsimulation device 400.

Control device 100 acquires teaching data from simulation device 400,and outputs to robot controller 300 a movement command as a function ofthe acquired teaching data. This allows robot controller 300 to operaterobot 200 according to the teaching data.

As noted above, the teaching data is created by the offline teachingthat utilizes the robot model on the virtual space. An error between thevirtual environment and the actual environment causes an error betweenthe actual operation of robot 200 and a simulation operation. Controlsystem SYS according to the present embodiment performs the followingsteps (1) through (6) in order to inhibit such an error from occurring.

(1) Simulation device 400 performs the offline teaching and one or moreteaching points are set.

(2) Control device 100 outputs an image capture command for acalibration image to camera controller 600, and acquires a calibrationimage of robot 200 captured by imaging device 500. Based on thecalibration image, control device 100 estimates a position and pose ofimaging device 500 relative to robot 200.

(3) Simulation device 400 arranges a robot model in the virtual space atone teaching point selected from the teaching data, and generates afirst simulation image of the robot model captured by the virtual camerathat is arranged so as to meet the following conditions A.

Conditions A: position and pose of the virtual camera relative to therobot model coincide with the position and pose of imaging device 500estimated in (2).

(4) Control device 100 generates a movement command (hereinafter,referred to as a “first movement command”) to the selected one teachingpoint, and outputs the first movement command to robot controller 300.

(5) After robot controller 300 drives robot 200 according to the firstmovement command, control device 100 outputs an image capture command tocamera controller 600. Control device 100 then acquires a first actualimage of robot 200 captured by imaging device 500 from camera controller600.

FIG. 2 is a diagram showing one example of an image obtained byoverlying the first actual image and the first simulation image one onthe other. First actual image 70 is an image of robot 200 that iscaptured after robot 200 has been driven according to the first movementcommand to the teaching point. First simulation image 80 is an imagecaptured of a robot model 90 that is arranged at the teaching point.Note that FIG. 2 shows a schematic depiction of robot 200 and robotmodel 90. Depending on an error between the actual environment and thevirtual environment, a deviation can occur between the position and poseof robot 200 on first actual image 70 and the position and pose of robotmodel 90 on first simulation image 80. Desirably, robot 200 performs thesame operates as robot model 90. To that end, the following step (6) isperformed.

(6) Control device 100 determines an amount of correction of theposition and pose of robot 200 for the teaching point so that theposition and pose of robot 200 on first actual image 70 approximates tothe position and pose of robot model 90 on first simulation image 80.

This allows the actual operation of robot 200 to be approximated to theoperation of robot model 90 on the virtual space simulated in simulationdevice 400. In other words, an error between the actual operation ofrobot 200 and the simulation operation, due to an error between theactual environment and the virtual environment, can be correctedautomatically. As a result, robot 200 can be accurately operatedaccording to simulation.

Note that if the teaching data describes multiple teaching points, thesteps (3) through (6) are repeated for each teaching point.

§ 2 Specific Example

Next, a specific example of control system SYS according to the presentembodiment will be described.

<A. Example Hardware Configuration of Control Device>

FIG. 3 is a block diagram showing an example hardware configuration of acontrol device according to the present embodiment. As shown in FIG. 3,control device 100 according to the present embodiment includes aprocessor 102, such as a central processing unit (CPU) or amicro-processing unit (MPU), a chipset 104, a main memory 106, a storage108, an upper-network controller 110, a universal serial bus (USB)controller 112, a memory card interface 114, and a field networkcontroller 120.

Processor 102 reads various programs from storage 108, deploy them intomain memory 106 and executes them. Chipset 104 controls data transferbetween processor 102 and each component.

Storage 108 stores a system program 136 for implementing a basicfunction, a user program 130, an estimation program 132, and acorrection program 134. User program 130 is a program for controlling anobject to be controlled. User program 130 is created by a user,depending on the purposes of the control. Estimation program 132 is aprogram for estimating the position and pose of imaging device 500relative to robot 200. Correction program 134 is a program forcorrecting the teaching data.

Upper-network controller 110 controls data exchange with other devicevia an upper network. USB controller 112 controls data exchange with theother device (e.g., simulation device 400) via USB connection. Note thatin the example shown in FIG. 3, data is exchanged between USB controller112 and simulation device 400. However, upper network controller 110 maycontrol the data exchange with simulation device 400 via the uppernetwork.

Memory card interface 114 is detachable from a memory card 116. Memorycard interface 114 is capable of writing data to memory card 116 andreading various data (user program 130, trace data, etc.) from memorycard 116.

Field network controller 120 controls data exchange with robotcontroller 300 and camera controller 600 via the field network.

While FIG. 3 illustrates a configuration in which necessary functionsare provided by processor 102 executing the programs, some or all thesefunctions provided may be implemented using a dedicated hardware circuit(e.g., ASIC (Application Specific Integrated Circuit) or FPGA(Field-Programmable Gate Array), etc.). Alternatively, the primarycomponents of control device 100 may be implemented using hardwareaccording to a general-purpose architecture (e.g., a general-purposepersonal computer-based industrial personal computer). In this case,using the virtualization technology, multiple operating systems fordifferent applications may be run in parallel and necessary applicationsmay be executed on the respective operating systems.

<B. Example Hardware Configuration of Simulation Device>

FIG. 4 is a schematic diagram showing an example hardware configurationof the simulation device according to the present embodiment. As oneexample, simulation device 400 is implemented using hardware (e.g., ageneral-purpose personal computer) according to a general-purposearchitecture.

As shown in FIG. 4, simulation device 400 includes a processor 402 suchas CPU or GPU, a main memory device 404, an input unit 406, a display408, a secondary memory device 410, an optical drive 412, and acommunication interface 418. These components are connected together viaa processor bus 420.

Processor 402 reads programs (as one example, an operating system (OS)430 and a simulation program 432) from secondary memory device 410,deploys them into main memory device 404 and executes them, therebyimplementing various processes.

In addition to OS 430 for implementing the basic functions, secondarymemory device 410 stores simulation program 432 for providing functionsas simulation device 400. Simulation program 432 implements simulationdevice 400 according to the present embodiment by being executed by aninformation processing device (substantially, processor 402) which is acomputer.

Secondary memory device 410 also stores robot CAD data 434 and objectCAD data 436. Robot CAD data 434 is CAD data which indicates the shapeof robot 200, indicating robot model 90 (see FIG. 2). Object CAD data436 is CAD data which indicates the shape of an object to be processedby robot 200, indicating an object model.

Input unit 406 is configured of a keyboard, a mouse, etc., and receivesuser manipulations. Display 408 displays, for example, a result ofprocessing from processor 402.

Communication interface 418 exchanges data with control device 100 viaany communication medium, such as a USB.

Simulation device 400 has optical drive 412. A program, which is storedin a recording medium 414 (e.g., an optical recording medium, such asDVD (Digital Versatile Disc)) storing computer-readable instructions ina non-transitory manner, is read and installed into, for example,secondary memory device 410.

While simulation program 432, which is executed by simulation device400, may be installed via computer-readable recording medium 414,simulation program 432 may be installed by downloading it from, forexample, a server device on a network. The functions provided bysimulation device 400 according to the present embodiment may beimplemented in a manner that utilizes a part of the module provided bythe OS.

While FIG. 4 shows the configuration example in which the functionsnecessary as simulation device are provided 400 by processor 402executing the programs, some or all of these functions provided may beimplemented using a dedicated hardware circuit (e.g., ASIC or FPGA).

<C. Example Functional Configuration of Simulation Device>

FIG. 5 is a block diagram showing an example functional configuration ofthe simulation device according to the present embodiment. As shown inFIG. 5, simulation device 400 includes a setting unit 42 and an imagegenerator 44. Setting unit 42 and image generator 44 are implemented byprocessor 402 executing simulation program 432.

Setting unit 42 sets teaching points by performing offline teachingaccording to user input.

FIG. 6 is a diagram illustrating a method of setting teaching points. Asshown in FIG. 6, setting unit 42 displays on display 408 (see FIG. 4) avirtual space in which robot model 90 indicated by robot CAD data 434(see FIG. 4) and an object model 95 indicated by object CAD data 436 arearranged. Robot model 90 includes a base model 91 corresponding to base21 (see FIG. 1), an arm model 92 corresponding to arm 22, and an endeffector model 93 corresponding to end effector 23.

The user is allowed to operate input unit 406 (see FIG. 4) to causerobot model 90 to perform a desired process (e.g., picking) on objectmodel 95 on the virtual space. The user causes robot model 90 to operateuntil end effector model 93 reaches the position and pose to be taken toperform the desired process, after which the user inputs a teachingpoint setting indication. Setting unit 42 sets the teaching pointsaccording to a state of robot model 90 at the time the teaching pointsetting indication has been input.

The teaching point, for example, indicates the position and pose of endeffector model 93 in a base model coordinate system BM referenced tobase model 91. The position of end effector model 93 is indicated by,for example, coordinate values (x, y, z) of the center of gravity G ofend effector model 93. The pose of end effector model 93 is indicated byrotation parameters (θx, θy, θz) represented by an Euler angle or afixed angle.

Returning to FIG. 5, setting unit 42 generates teaching data describingthe set one or more teaching points in chronological order, and outputsthe generated teaching data to control device 100. In the example shownin FIG. 5, the n-th teaching point Pn is indicated by (xn, yn, zn, θxn,θyn, θzn).

Image generator 44 generates a simulation image captured of robot model90 by the virtual camera, disposed arranged in the virtual space.

Image generator 44 receives from control device 100 the informationindicating the position and pose of imaging device 500 relative to robot200 (e.g., a transformation matrix ^(B)H_(C)). Transformation matrix^(B)H_(C) is a matrix for transforming the base coordinate systemreferenced to base 21 of robot 200 into a camera coordinate systemreferenced to imaging device 500.

Image generator 44 generates a first simulation image of the robot model90 for each teaching point set by setting unit 42. The first simulationimage is of robot model 90 arranged at the teaching point, captured bythe virtual camera that is arranged on the virtual space so that theposition and pose of the virtual camera relative to robot model 90coincide with the position and pose of imaging device 500 relative torobot 200 indicated by transformation matrix ^(B)H_(C).

FIG. 7 is a diagram illustrating a method of generation of a firstsimulation image corresponding to the n-th teaching point Pn. As shownin FIG. 7, image generator 44 moves robot model 90 so that the positionand pose of end effector model 93 is at teaching point Pn.

Furthermore, image generator 44 arranges virtual camera 94 so that atransformation matrix ^(BM)H_(VC) coincides with transformation matrix^(B)H_(C) received from control device 100, the transformation matrix^(BM)H_(VC) being for transforming base model coordinate system BMreferenced to base model 91 into a virtual camera coordinate system CMreferenced to a virtual camera 94. Transformation matrix ^(BM)H_(VC)represents components of a basis vector and the position of origin forthe virtual camera coordinate system in terms of the base modelcoordinate system, showing the position and pose of virtual camera 94relative to robot model 90 (specifically, base model 91).

Returning to FIG. 5, image generator 44 output to control device 100first simulation image 80 generated for each teaching point. Firstsimulation image 80 contains robot model 90 arranged at the teachingpoint, as viewed from virtual camera 94.

<D. Example Functional Configuration of Control Device>

FIG. 8 is a block diagram showing an example functional configuration ofthe control device according to the present embodiment. As shown in FIG.8, control device 100 includes an estimation unit 11, a firstacquisition unit 12, and a determination unit 13. Estimation unit 11 isimplemented by processor 102 executing estimation program 132 shown inFIG. 3. First acquisition unit 12 and determination unit 13 areimplemented by processor 102 executing correction program 134 shown inFIG. 3.

<D-1. Estimation Unit>

Estimation unit 11 estimates the position and pose of imaging device 500relative to robot 200, based on an image of robot 200 captured byimaging device 500. Estimation unit 11 may utilize a well-knowntechnology to estimate the position and pose of imaging device 500relative to robot 200.

For example, estimation unit 11 utilizes a well-known solution of thePerspective-n-Point problem to estimate the position and pose of imagingdevice 500 relative to robot 200. The solution of thePerspective-n-Point problem is disclosed in, for example, “A UnifiedSolution to the PnP Problem for General Camera Model Based on DirectComputation of All the Stationary Points,” by Gaku NAKANO and twoothers, IEICE Transaction D Vol. J95-D, No. 8, pp. 1565-1572, 2012.

FIG. 9 is a diagram illustrating one example of the method of estimationof the position and pose of the imaging device relative to the robot.Three-dimensional coordinates for at least three feature points A1 to Am(m is an integer greater than or equal to 3) on base 21 set at the fixedposition are predetermined and recorded to control device 100. Featurepoints A1 to Am, for example, indicates edges of base 21. Thethree-dimensional coordinates of feature points A1 to An are indicatedby the base coordinate system B referenced to base 21.

Estimation unit 11 acquires from imaging device 500 a calibration imageof robot 200 captured by imaging device 500. Estimation unit 11 detectsfeature points A1 to Am in the calibration image and determinescoordinates of feature points A1 to Am on the calibration image.

Estimation unit 11 solves the Perspective-n-Point problem using thethree-dimensional coordinates of feature points A1 to Am on the basecoordinate system and the coordinates of feature points A1 to Am on thecalibration image, thereby estimating position and pose of imagingdevice 500 relative to robot 200. The position and pose of imagingdevice 500 relative to robot 200 are indicated by a transformationmatrix ^(B)H_(C) which transforms, for example, the base coordinatesystem B into a camera coordinate system C (a coordinate systemreferenced to imaging device 500).

Note that the method of estimation of the position and pose of imagingdevice 500 relative to robot 200 is not limited thereto.

For example, estimation unit 11 may use an image acquired from imagingdevice 500 and an image acquired from an imaging device that isseparately disposed from imaging device 500, to estimate the positionand pose of imaging device 500 relative to robot 200. The method ofestimation, using two cameras, of the position and pose of a camera mayutilize, for example, the technology disclosed in “Camera Position andPose Estimation 0, Epipolar Geometry,” [online], Jan. 26, 2018, DailyTech Blog, [Searched on Jun. 1, 2019], the Internet<http://daily-tech.hatenablog.com/entry/2018/01/26/064603>.

Alternatively, estimation unit 11 may use multiple images of robot 200that are captured while moving the imaging device 500 to estimate theposition and pose of imaging device 500 relative to robot 200. Themethod of estimation, using mobile camera images, of the position andpose of a camera may utilize, for example, the technology disclosed in“3D Shape Reconstruction from Mobile Camera Images and Localization(SLAM) and Dense 3D shape reconstruction” by Akihito SEKI, IPSJ SIGTechnical Report, January, 2014, Vol. 2014-CVIM-190, No. 40.

Alternatively, estimation unit 11 may use multiple images of robot 200that are captured while moving end effector 23 having calibrationmarkers attached thereon, to estimate the position and pose of imagingdevice 500 relative to robot 200. This estimation method is calledhand-eye calibration. The method of estimation of the position and poseof the camera using the hand-eye calibration may utilize, for example,the technology disclosed in “Development of Teaching Methods for theRehabilitation Robot” [online], 2009, Technical Report by Aichi Centerfor Industry and Science Technology, [Searched on Jun. 1, 2019], theInternet <http://www.aichi-inst.jp/sangyou/research/report/2009_01.pdf>.

Estimation unit 11 outputs the information indicating the position andpose of imaging device 500 relative to robot 200 (e.g., transformationmatrix ^(B)H_(C)) to simulation device 400.

<D-2. First Acquisition Unit>

For each teaching point indicated by the teaching data, firstacquisition unit 12 generates a first movement command to the teachingpoint, and provides robot controller 300 with the first movementcommand. Robot 200 is driven according to the first movement command,after which the estimation unit 11 outputs an image capture command tocamera controller 600. In this way, estimation unit 11 acquires firstactual image 70 of robot 200 that is captured after robot 200 has beendriven in response to the first movement command.

<D-3. Determination Unit>

Determination unit 13 determines an amount of correction of the positionand pose of robot 200 for the teaching point so that the position andpose of robot 200 on first actual image 70 approximates to the positionand pose of robot model 90 on first simulation image 80.

As shown in FIG. 8, determination unit 13 has a deviation calculationunit 14, a second acquisition unit 15, and a correction amountcalculation unit 16.

(D-3-1. Deviation Calculation Unit)

Deviation calculation unit 14 calculates the deviation between theposition and pose of robot 200 on the actual image and the position andpose of robot model 90 on first simulation image 80.

Deviation calculation unit 14 overlays the actual image and firstsimulation image 80 one on the other, and calculates the distancebetween the feature point of robot 200 on the actual image and thefeature point of robot model 90 on first simulation image 80 as thedeviation.

In the example shown in FIG. 2, feature points B1, B2, which are thedistal ends of two claws of end effector 23, and a feature point B3,which is the coupling of the two claws, are extracted as feature pointsof robot 200. Similarly, feature points C1, C2, which are the distalends of two claws of end effector model 93, and a feature point C3,which is the coupling of the two claws, are extracted as feature pointsof robot model 90. For example, deviation calculation unit 14 calculatesthe sum of distances from feature points B1, B2, and B3 to featurepoints C1, C2, and C3, respectively, as the deviation between theposition and pose of robot 200 on the actual image and the position andpose of robot model 90 on first simulation image 80.

Deviation calculation unit 14 calculates the deviation between theposition and pose of robot 200 on first actual image 70 acquired byfirst acquisition unit 12 and the position and pose of robot model 90 onfirst simulation image 80, and the deviation between a second actualimage 72 acquired by second acquisition unit 15 described below and theposition and pose of robot model 90 on first simulation image 80. In thefollowing, the deviation between the position and pose of robot 200 onfirst actual image 70 and the position and pose of robot model 90 onfirst simulation image 80 will be referred to as a “first deviation.”The deviation between the position and pose of robot 200 on secondactual image 72 and the position and pose of robot model 90 on firstsimulation image 80 will be referred to as a “second deviation.”

(D-3-2. Second Acquisition Unit)

Second acquisition unit 15 performs the following acquisition process inresponse to the first deviation being greater than or equal to apredetermined threshold. In other words, second acquisition unit 15generates a second movement command in a direction that reduces thedeviation between the position and pose of robot 200 on the actual imageand the position and pose of robot model 90 on first simulation image80, and provides robot controller 300 with the second movement command.Robot 200 is driven in response to the second movement command, afterwhich the second acquisition unit 15 outputs an image capture command tocamera controller 600. Second acquisition unit 15 then acquires secondactual image 72 of robot 200 that is captured after robot 200 has beendriven in response to the second movement command.

Second acquisition unit 15 repeatedly performs the above acquisitionprocess until the second deviation between the position and pose ofrobot 200 on second actual image 72 and the position and pose of robotmodel 90 on first simulation image 80 is less than the threshold.

Second acquisition unit 15 may utilize a well-known visual servoingtechnique to generate the second movement command. The visual servoingtechnique is disclosed in, for example, “Visual Servoing,” Measurementand Control, vol. 35, no. 4, p. 282-285, April, 1996 by KoichiHASHIMOTO.

FIG. 10 is a diagram showing one example of changes in the actual imagewhen the acquisition process is performed. FIG. 10 shows first actualimage 70, second actual image 72_1 acquired by the first iteration ofthe acquisition process, and second actual image 72_k acquired by thek-th iteration of the acquisition process (k is an integer greater thanor equal to 2). As shown in FIG. 10, the first deviation correspondingto first actual image 70 is large. The second deviation corresponding tosecond actual image 72_1 is less than the first deviation. The seconddeviation corresponding to second actual image 72_k is nearly zero.

(D-3-3. Correction Amount Calculation Unit)

Correction amount calculation unit 16 calculates an amount of movementfrom the position and pose of robot 200 having been driven according tothe first movement command to the position and pose of robot 200 whenthe second deviation is less than the threshold (hereinafter, referredto as a “necessarily amount of movement”), as the amount of correctionof the position and pose of robot 200 for the teaching point. Correctionamount calculation unit 16 may calculate the cumulative amount of secondmovement commands provided to robot controller 300 by second acquisitionunit 15 as the necessarily amount of movement of robot 200.

The necessarily amount of movement is indicated by translation amountsdx, dy, dz and rotational movement amounts dθx, dθy, dθz in the basecoordinate system. Correction amount calculation unit 16 may correctteaching point P by adding the necessarily amount of movement (theamount of correction) to coordinate values (x, y, z, θx, θy, θz)indicative of teaching point P. In other words, the corrected teachingpoint is indicated by (x+dx, y+dy, z+dz, θx+dθx, θy+dθy, θz+dθz).

<E. Flow of Correction Amount Determination Process>

Referring to FIGS. 11 and 12, a flow of the correction amountdetermination process in control system SYS will be described. FIG. 11is a flow chart illustrating a flow of the first half of the process fordetermining an amount of correction of the position and pose of robot200. FIG. 12 is a flow chart illustrating a flow of the last half of theprocess for determining the amount of correction.

Initially, processor 402 included in simulation device 400 performsoffline teaching, thereby setting teaching points (step S1). Processor402 generates teaching data describing the set N teaching points inchronological order, and records the generated teaching data to storage108 (step S2). The teaching data recorded to storage 108 is output tocontrol device 100.

Next, processor 102 included in control device 100 outputs an imagecapture command to camera controller 600 (step S3). Imaging device 500captures an image of robot 200 according to control by camera controller600 (step S4). Camera controller 600 transmits to control device 100 theimage (a calibration image) captured by imaging device 500 (step S5).

Based on the calibration image, processor 102 included in control device100 estimates the position and pose of imaging device 500 relative torobot 200 (step S6). Processor 102 notifies simulation device 400 of theestimated position and pose of imaging device 500 (step S7).

Processor 402 included in simulation device 400 arranges the virtualcamera on the virtual space so that the position and pose of the virtualcamera relative to robot model 90 coincide with the position and pose ofimaging device 500 relative to robot 200 notified from control device100 (step S8). For each of N teaching points, processor 402 generatesfirst simulation image 80 of the robot model 90 when robot model 90 isarranged at the teaching point (step S9). Processor 402 transfers thegenerated first simulation images 80 to control device 100 (step S10).

Next, processor 102 included in control device 100 executes a first loopof steps S11 through S28. In other words, processor 102 sets n to 1through N in order and executes the first loop for calculating theamount of correction for the n-th teaching point.

In step S12, processor 102 generates a first movement command to then-th teaching point, and provides robot controller 300 with the firstmovement command. Robot controller 300 computes a trajectory from thecurrent position and pose of robot 200 to the n-th teaching point (stepS13). Robot controller 300 outputs to robot 200 a target value commandaccording to the computed trajectory (step S14). Robot controller 300acquires from robot 200 the information indicating the current positionand pose of robot 200 (step S15). In this way, robot controller 300confirms that the position and pose of robot 200 has reached theteaching point.

Next, processor 102 included in control device 100 executes a secondloop of steps S16 through S26. Processor 102 repeats the second loopuntil the deviation between the position and pose of robot 200 on theactual image (first actual image 70 or second actual image 72) and theposition and pose of robot model 90 on first simulation image 80 is lessthan the threshold.

In step S17, processor 102 outputs an image capture command to cameracontroller 600. Imaging device 500 captures an image of robot 200according to control by camera controller 600 (step S18). Cameracontroller 600 transmits the actual image of robot 200 captured byimaging device 500 to control device 100 (step S19). Note that theactual image obtained in step S19 of the first iteration of the secondloop is first actual image 70 of robot 200 that is captured after robot200 has been driven in response to the first movement command.

Processor 102 overlays the actual image, received in step S19, and firstsimulation image 80 one on the other (step S20). Next, processor 102calculates the deviation between the position and pose of robot 200 onthe actual image and the position and pose of robot model 90 on firstsimulation image 80 and determines whether the calculated deviation isless than the threshold (step S21).

If the deviation is not less than the threshold (NO in step S21),processor 102 generates a movement command (the second movement command)in a direction that reduces the deviation, and provides robot controller300 with the second movement command (step S22). Robot controller 300computes a trajectory from the current position and pose of robot 200 tothe position and pose that corresponds to the second movement command(step S23). Robot controller 300 outputs to robot 200 a target valuecommand according to the computed trajectory (step S24). Robotcontroller 300 acquires from robot 200 the information indicating thecurrent position and pose of robot 200 (step S25). In this way, robotcontroller 300 confirms that robot 200 has reached the position and posecorresponding to the second movement command. Steps S17 through S21 arethen repeated. At this time, the actual image obtained in step S19 issecond actual image 72 of robot 200 that is captured after robot 200 hasbeen driven in response to the second movement command.

If the deviation is less than the threshold (YES in step S21), processor102 ends the second loop, and, in step S27, calculates the cumulativeamount of the second movement commands (the cumulative amount ofmovements) as the amount of correction for the n-th teaching point.

In this way, the first loop (steps S16 through S28) is executed for n=1through N and the amounts of correction for the first teaching pointthrough the n-th teaching point are thereby determined.

<F. Variations>

<F-1. Variation 1>

In the above description, robot 200 is moved so that the position andpose of robot 200 on the actual image coincide with the position andpose of robot model 90 on first simulation image 80, and this amount ofmovement of robot 200 is determined as an amount of correction of theposition and pose of robot 200 for the teaching point. According to thismethod, the actual operation of robot 200 can be accurately conformed tothe simulation operation. However, since robot 200 is moved, it takestime to determine the amount of correction.

Thus, in order to reduce the time it takes to determine the amount ofcorrection, according to control system SYS of Variation 1, robot model90 is moved so as to conform to the position and pose of robot 200 onfirst actual image 70. In the following, details of the control systemSYS according to Variation 1 will be described.

Deviation calculation unit 14 according to Variation 1 calculates thedeviation between the position and pose of robot 200 on first actualimage 70 and the position and pose of robot model 90 on a simulationimage. Deviation calculation unit 14 calculates the deviation betweenthe position and pose of robot 200 on first actual image 70 and theposition and pose of robot model 90 on the above first simulation image80 and the deviation between the position and pose of robot 200 on firstactual image 70 and the position and pose of robot model 90 on a secondsimulation image described below. In the following, the deviationbetween the position and pose of robot 200 on first actual image 70 andthe position and pose of robot model 90 on the second simulation imagewill be referred to as a “third deviation.” Note that the deviationbetween the position and pose of robot 200 on first actual image 70 andthe position and pose of robot model 90 on first simulation image 80will be referred to as a “first deviation” also in Variation 1.

In response to the first deviation being equal to or greater than thethreshold, image generator 44 moves robot model 90 in a direction thatreduces the deviation between the position and pose of robot 200 onfirst actual image 70 and the position and pose of robot model 90 on thesimulation image. Image generator 44 stores the amount of movement ofrobot model 90. Image generator 44 then generates the second simulationimage of robot model 90 captured by virtual camera 94.

Correction amount calculation unit 16 calculates the amount of movement(necessarily amount of movement) from the position and pose of robotmodel 90 when the third deviation is less than the threshold to theposition and pose of robot model 90 arranged at the teaching point, asthe amount of correction of the position and pose of robot 200 for theteaching point.

FIG. 13 is a flow chart illustrating a flow of a part of the process fordetermining the amount of correction according to Variation 1. Thecorrection amount determination process according to Variation 1 is thesame as the correction amount determination process illustrated in FIGS.11 and 12, except that the correction amount determination processaccording to Variation 1 includes steps S31 through S33, instead ofsteps S22 through S25, and step S34, instead of step S27.

If NO in step S21, processor 402 included in simulation device 400 movesrobot model 90 in a direction that reduces the deviation between theposition and pose of robot 200 on first actual image 70 and the positionand pose of robot model 90 on the simulation image (step S31).

Next, processor 402 generates a second simulation image of robot model90 captured by virtual camera 94 (step S32). Processor 402 transfers thegenerated second simulation image to control device 100 (step S33).After step S33, steps S20, S21 are repeated.

If YES in step S21, processor 102 ends the second loop. Processor 102then calculates an amount of movement from the position and pose ofrobot model 90 when the deviation is less than the threshold to theposition and pose of robot model 90 disposed at the teaching point, asan amount of correction for the teaching point (step S34).

According to Variation 1, the steps for moving robot 200 (steps S22through S25 in FIG. 12) are unnecessary, thereby reducing the time ittakes to determine the amount of correction. However, due to an errorbetween the actual environment and the virtual environment, the amountof correction calculated in step S34 has lower accuracy than the amountof correction calculated in step S27. For this reason, control systemSYS according to Variation 1 is applicable to the case where an errorbetween the actual operation of robot 200 and the simulation operationis allowed to some extent.

<F-2. Variation 2>

In the above description, control device 100 determines the amount ofcorrection of the position and pose of robot 200 for the teaching pointaccording to comparison of a two-dimensional actual image with asimulation image. However, control device 100 may determine the amountof correction for the teaching point according to comparison of athree-dimensional actual image with a simulation image.

For example, according to a three-dimensional stereo measurement, cameracontroller 600 may generate a three-dimensional point cloud (athree-dimensional image) based on two-dimensional images acquired fromtwo imaging devices. Alternatively, according to a patterned lightprojection method, camera controller 600 may generate athree-dimensional point cloud based on patterned light on atwo-dimensional image. As a simulation image, image generator 44included in simulation device 400 may generate a three-dimensional pointcloud (a three-dimensional image) as viewed from virtual camera 94.

According to Variation 2, the actual operation of robot 200 can beaccurately conformed to the simulation operation.

<F-3. Variation 3>

In the above description, estimation unit 11 is included in controldevice 100. However, estimation unit 11 may be included in cameracontroller 600.

<G. Operations and Effects>

As described above, control system SYS according to the presentembodiment includes simulation device 400, robot controller 300, andimaging device 500. Simulation device 400 simulates the operation ofrobot 200, using robot model 90 indicating the shape of robot 200. Robotcontroller 300 drives robot 200 according to a provided movementcommand. Imaging device 500 captures an image of robot 200. Controlsystem SYS further includes estimation unit 11 (processor 102) thatestimates the position and pose of imaging device 500 relative to robot200 based on the image captured by imaging device 500. Simulation device400 includes setting unit 42 and image generator 44. Setting unit 42performs offline teaching, thereby setting a teaching point indicativeof the position and pose to be taken by robot 200. Image generator 44generates a simulation image of robot model 90 captured by virtualcamera 94 that is arranged in a virtual space. Image generator 44arranges robot model 90 at the teaching point in the virtual space andgenerates first simulation image 80 of robot model 90 captured byvirtual camera 94 that is arranged so that the position and pose ofvirtual camera 94 relative to robot model 90 coincide with the positionand pose of imaging device 500 relative to robot 200 estimated byestimation unit 11. Control system SYS further includes firstacquisition unit 12 and determination unit 13. First acquisition unit 12provides robot controller 300 with a first movement command to theteaching point, and acquires first actual image 70 of robot 200 that iscaptured by imaging device 500 after robot 200 has been driven accordingto the first movement command. Determination unit 13 determines anamount of correction of the position and pose of robot 200 for theteaching point so that the position and pose of robot 200 on firstactual image 70 approximates to the position and pose of robot model 90on first simulation image 80.

According to the above configuration, the actual operation of robot 200can be approximated to the operation of robot model 90 on the virtualspace simulated in simulation device 400. In other words, an errorbetween the actual operation of robot 200 and the simulation operation,due to an error between the actual environment and the virtualenvironment, can be corrected automatically. As a result, robot 200 canbe accurately operated according to simulation.

Moreover, imaging device 500 is installed in the actual environment,after which the estimation unit 11 estimates the position and pose ofimaging device 500 relative to robot 200. Virtual camera 94 is thenarranged so that the position and pose of virtual camera 94 relative torobot model 90 coincide with the position and pose of imaging device 500relative to robot 200 estimated by estimation unit 11. In other words,the position and pose of imaging device 500 is reflected to thesimulated position and pose of virtual camera 94. This obviates the needfor precisely adjusting the position and pose of imaging device 500.

Determination unit 13 includes deviation calculation unit 14 andcorrection amount calculation unit 16. Deviation calculation unit 14 maycalculate the deviation between the position and pose of robot 200 onthe actual image captured by imaging device 500 and the position andpose of robot model 90 on first simulation image 80. Correction amountcalculation unit 16 may calculate the amount of movement from theposition and pose of robot 200 having been driven according to the firstmovement command to the position and pose of robot 200 when thedeviation is less than a predetermined threshold, as the amount ofcorrection of the position and pose of robot 200.

According to the above configuration, the actual operation of robot 200can be more accurately conformed to the simulation operation.

Specifically, determination unit 13 may include second acquisition unit15. Second acquisition unit 15 provides robot controller 300 with asecond movement command in a direction that reduces the deviation, andperforms the acquisition process to acquire second actual image 72 ofrobot 200 that is captured by imaging device 500 after robot 200 hasbeen driven according to the second movement command. Second acquisitionunit 15 repeatedly performs the acquisition process since robot 200 hasbeen driven according to the first movement command until the deviationis less than the threshold. Correction amount calculation unit 16 maycalculate the cumulative amount of the second movement commands providedto robot controller 300 as the amount of correction.

This facilitates the calculation of the amount of movement from theposition and pose of robot 200 having been driven according to the firstmovement command to the position and pose of robot 200 when thedeviation is less than a predetermined threshold.

Deviation calculation unit 14 may calculate the deviation between theposition and pose of robot 200 on first actual image 70 and the positionand pose of robot model 90 on a simulation image (first simulation image80 and second simulation image 82). Correction amount calculation unit16 may then calculate the amount of movement from the position and poseof robot model 90 when the deviation is less than the predeterminedthreshold to the position and pose of robot model 90 arranged at theteaching point, as the amount of correction.

This obviates the need for moving robot 200 when determining the amountof correction, thereby reducing the time it takes to determine theamount of correction.

Control system SYS further includes control device 100. Control device100 is connected to imaging device 500, robot controller 300, andsimulation device 400, and controls robot controller 300 and imagingdevice 500. Estimation unit 11, first acquisition unit 12, anddetermination unit 13 are included in control device 100.

According to the above configuration, control device 100 estimates theposition and pose of imaging device 500 relative to robot 200 anddetermines the amount of correction of the position and pose of robot200. Robot controller 300 performs trajectory calculation, etc. fordriving robot 200. Simulation device 400 performs offline teaching. Assuch, the calculations that are necessary to determine the amount ofcorrection for the teaching point are carried out in a distributedfashion among multiple pieces of equipment, including control device100, robot controller 300, and simulation device 400. In other words,loads on the equipment can be distributed.

<H. Additional Statements>

As described above, the present embodiment and variations thereofinclude the disclosure as follows:

(Configuration 1)

A control system (SYS) for controlling a robot (200), the control system(SYS) including:

a simulation device (400) for performing simulation using a robot model(90) indicating a shape of the robot (200);

a robot controller (300) for driving the robot (200) according to amovement command provided to the robot controller (300);

an imaging device (500) for capturing an image of the robot (200); and

an estimation module (11, 102) for estimating a position and pose of theimaging device (500) relative to the robot (200) based on the image ofthe robot (200) captured by the imaging device (500), wherein

the simulation device (400) includes:

a setting module (42, 402) for performing offline teaching to set ateaching point indicative of a position and pose to be taken by therobot (200); and

an image generating module (44, 402) for generating a simulation imageof the robot model (90) captured by a virtual camera (94) arranged in avirtual space, wherein

the image generating module (44, 402) arranges the robot model (90) atthe teaching point in the virtual space and generates a first simulationimage (80) of the robot model (90) captured by the virtual camera (94)that is arranged so that a position and pose of the virtual camera (94)relative to the robot model (90) coincide with the position and pose ofthe imaging device (500) estimated by the estimation module (11, 102),

the control system (SYS) further including:

a first acquisition module (12, 102) for providing the robot controller(300) with a first movement command to the teaching point and acquiringa first actual image (70) of the robot (200) captured by the imagingdevice (500) after the robot (200) has been driven according to thefirst movement command; and

a determination module (13, 102) for determining an amount of correctionof a position and pose of the robot (200) for the teaching point so thata position and pose of the robot (200) on the first actual image (70)approximates to a position and pose of the robot model (90) on the firstsimulation image (80).

(Configuration 2)

The control system (SYS) according to Configuration 1, wherein

the determination module (13, 102) includes:

a deviation calculation module (14, 102) for calculating a deviationbetween the position and pose of the robot (200) on an actual image ofthe robot (200) captured by the imaging device (500) and the positionand pose of the robot model (90) on the first simulation image (80); and

a correction amount calculating module (16, 102) for calculating, as theamount of correction, an amount of movement from the position and poseof the robot (200) that has been driven according to the first movementcommand to a position and pose of the robot (200) when the deviation isless than a predetermined threshold.

(Configuration 3)

The control system (SYS) according to Configuration 2, wherein

the determination module (13, 102) further includes

a second acquisition module (15, 102) for providing the robot controller(300) with a second movement command in a direction that reduces thedeviation and performing an acquisition process for acquiring a secondactual image captured of the robot (200) by the imaging device (500)when the robot (200) has been driven according to the second movementcommand, wherein

the second acquisition module (15, 102) repeatedly performs theacquisition process since the robot (200) has been driven according tothe first movement command until the deviation is less than thethreshold, and

as the amount of correction, the correction amount calculating module(16, 102) calculates a cumulative amount of second movement commandsprovided to the robot controller (300).

(Configuration 4)

The control system (SYS) according to Configuration 1, wherein

the determination module (13, 102) includes:

a deviation calculation module (14, 102) for calculating a deviationbetween the position and pose of the robot (200) on the first actualimage (70) and a position and pose of the robot model (90) on thesimulation image; and

a correction amount calculating module (16, 102) for calculating, as theamount of correction, an amount of movement from a position and pose ofthe robot model (90) when the deviation is less than a predeterminedthreshold to a position and pose of the robot model (90) arranged at theteaching point.

(Configuration 5)

The control system (SYS) according to any one of Configurations 1 to 3,including

a control device (100) that controls the robot controller (300) and theimaging device (500), the control device (100) being connected to theimaging device (500), the robot controller (300), and the simulationdevice (400), wherein

the estimation module (11, 102), the first acquisition module (12, 102),and the determination module (13, 102) are included in the controldevice (100).

(Configuration 6)

A control method in a control system (SYS) for controlling a robot(200), the control system (SYS) including:

a robot controller (300) for driving the robot (200) according to amovement command provided to the robot controller (300); and

an imaging device for capturing an image of the robot (200), the controlmethod including:

performing offline teaching using a robot model (90) to set a teachingpoint indicating a position and pose to be taken by the robot (200);

estimating a position and pose of the imaging device (500) relative tothe robot (200) based on the image of the robot (200) captured by theimaging device (500);

arranging the robot model (90) at the teaching point in a virtual spaceand generating a simulation image (80) of the robot model (90) capturedby a virtual camera (94) that is arranged so that a position and pose ofthe virtual camera (94) relative to the robot model (90) coincide withthe estimated position and pose of the imaging device (500);

providing the robot controller (300) with a movement command to theteaching point and acquiring an actual image (70) of the robot (200)captured by the imaging device (500) after the robot (200) has beendriven according to the movement command to the teaching point; and

determining an amount of correction of a position and pose of the robot(200) for the teaching point so that a position and pose of the robot(200) on the actual image (70) approximates to a position and pose ofthe robot model (90) on the simulation image (80).

Although the present invention has been described and illustrated indetail, it is clearly understood that the same is by way of illustrationand example only and is not to be taken by way of limitation, the scopeof the present invention being interpreted by the terms of the appendedclaims.

What is claimed is:
 1. A control system for controlling a robot, thecontrol system comprising: a simulation device for performing simulationusing a robot model indicating a shape of the robot; a robot controllerfor driving the robot according to a movement command provided to therobot controller; an imaging device for capturing an image of the robot;and an estimation module for estimating a position and pose of theimaging device relative to the robot based on the image of the robotcaptured by the imaging device, wherein the simulation device includes:a setting module for performing offline teaching to set a teaching pointindicative of a position and pose to be taken by the robot; and an imagegenerating module for generating a simulation image of the robot modelcaptured by a virtual camera arranged in a virtual space, wherein theimage generating module arranges the robot model at the teaching pointin the virtual space and generates a first simulation image of the robotmodel captured by the virtual camera that is arranged so that a positionand pose of the virtual camera relative to the robot model coincide withthe position and pose of the imaging device estimated by the estimationmodule, the control system further comprising a first acquisition modulefor providing the robot controller with a first movement command to theteaching point and acquiring a first actual image of the robot capturedby the imaging device after the robot has been driven according to thefirst movement command; and a determination module for determining anamount of correction of a position and pose of the robot for theteaching point so that a position and pose of the robot on the firstactual image approximates to a position and pose of the robot model onthe first simulation image.
 2. The control system according to claim 1,wherein the determination module includes: a deviation calculationmodule for calculating a deviation between the position and pose of therobot on an actual image of the robot captured by the imaging device andthe position and pose of the robot model on the first simulation image;and a correction amount calculating module for calculating, as theamount of correction, an amount of movement from the position and poseof the robot that has been driven according to the first movementcommand to a position and pose of the robot when the deviation is lessthan a predetermined threshold.
 3. The control system according to claim2, wherein the determination module further includes a secondacquisition module for providing the robot controller with a secondmovement command in a direction that reduces the deviation andperforming an acquisition process for acquiring a second actual imagecaptured of the robot by the imaging device when the robot has beendriven according to the second movement command, wherein the secondacquisition module repeatedly performs the acquisition process since therobot has been driven according to the first movement command until thedeviation is less than the threshold, and as the amount of correction,the correction amount calculating module calculates a cumulative amountof second movement commands provided to the robot controller.
 4. Thecontrol system according to claim 1, wherein the determination moduleincludes: a deviation calculation module for calculating a deviationbetween the position and pose of the robot on the first actual image anda position and pose of the robot model on the simulation image; and acorrection amount calculating module for calculating, as the amount ofcorrection, an amount of movement from a position and pose of the robotmodel when the deviation is less than a predetermined threshold to aposition and pose of the robot model arranged at the teaching point. 5.The control system according to claim 1, comprising a control devicethat controls the robot controller and the imaging device, the controldevice being connected to the imaging device, the robot controller, andthe simulation device, wherein the estimation module, the firstacquisition module, and the determination module are included in thecontrol device.
 6. A control method in a control system for controllinga robot, the control system including: a robot controller for drivingthe robot according to a movement command provided to the robotcontroller; and an imaging device for capturing an image of the robot,the control method comprising: performing offline teaching using a robotmodel to set a teaching point indicating a position and pose to be takenby the robot; estimating a position and pose of the imaging devicerelative to the robot based on the image of the robot captured by theimaging device; arranging the robot model at the teaching point in avirtual space and generating a simulation image of the robot modelcaptured by a virtual camera that is arranged so that a position andpose of the virtual camera relative to the robot model coincide with theestimated position and pose of the imaging device; providing the robotcontroller with a movement command to the teaching point and acquiringan actual image of the robot captured by the imaging device after therobot has been driven according to the movement command to the teachingpoint; and determining an amount of correction of a position and pose ofthe robot for the teaching point so that a position and pose of therobot on the actual image approximates to a position and pose of therobot model on the simulation image.